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R e- SEARCH! Re- SEARCH! 
Woo-hoo! Lets hear it for 
research! 

Okay, this is probably not some- 
thing you’d expect to hear at your 
local sports bar or even at NECC 
2005. Let’s face it: research is just not 
something that most people get excit- 
ed about. It can be complicated, ab- 
stract, and downright boring. When 
it comes to educational technology 
research, most of us would rather just 
have the bullet points — tell me what 
I need to do to make X or Y happen in 
my classroom. Or worse, some would 
rather not know anything about 
the latest research because we fear it 
might be contrary to the way we’ve 
been teaching for so long. 

Well, let’s all just take a deep 
breath, step back for a second, and 
relax. Research doesn’t have to be so 
intimidating, confusing, or boring. 
Perhaps it will help if you just un- 
derstand what the heck researchers 
are saying, what their terminology is. 
Let’s take this month’s Research Win- 
dows (RW) column to cover some of 
the common terms. 

Research Versus Evaluation 

What is the difference between re- 
search and evaluation? We hear about 
them quite often, and in today’s heav- 
ily grant-funded educational climate, 
most teachers and/or their students 
have been the subjects of either re- 
search or evaluation (or both) at one 
time or another. 

I can’t give you a definitive answer 
to this question, because it’s one that 
researchers and evaluators have been 
kicking around for decades. But I can 
give you a common perspective on 
the two. It’s important to note that 
research and evaluation are not neces- 
sarily the same thing. Evaluation has 
at its heart determining whether grant 
or project goals have been met. To the 
degree that those goals may not have 
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been met (yet), evaluators usually at- 
tempt to specify challenges and barri- 
ers to success. From that information, 
the grants principal investigator (i.e., 
the person in charge of the grant), 
the school principal, or the classroom 
teacher can make informed decisions 
about how to close the gaps in the 
current project and how to avoid such 
problems in the future. So, in this 
sense, evaluation equals action. 

The ultimate outcome of research 
is to understand what, how, and ulti- 
mately why. Researchers often begin 
with the exploration and description 
of a situation. But the quest for ex- 
planation, cause and effect, is what 
drives the researcher. In this sense, 
research equals knowledge. 

In addition, much empirical 
research — that is, research based on 
observation of the world and collect- 
ing data from that observation — 
relies on the concept of reliability, 
that what is studied in one research 
project could be repeated in other 
research projects given the same cir- 
cumstances. For example, if I were 
a university professor conducting re- 
search on the effects of socioeconomic 
status on math and reading test scores 
in Maryland, I would use a research 
design (a plan for the research) other 
researchers could replicate in other 
states with similar data. 

Evaluation is primarily concerned 
with the organizations own project 
and its success, but not so much 
with ensuring that the project could 
be replicated. This is not to say that 
evaluators use wacky research meth- 
ods that cant be used anywhere else. 
On the contrary, most evaluators are 
trained in common research methods 
and will use a research design that can 
be used elsewhere. But replicability is 
not the number-one goal. 

Now, these two paradigms — pure 
research and pure evaluation — are 
often in accord with one another. 


Evaluating a projects shortcomings 
in its goals will always require some 
understanding of why, of what caused 
a certain expected or unexpected out- 
come. Once the why is pinned down, 
hopefully wedl have a good idea of 
how to make changes. Likewise, re- 
search is often conducted with the 
eventual goal of inducing change. 

Okay, so next question: what kinds 
of research designs are commonly 
used? There are a lot of them, varying 
in their use from field to field, but 
let s stick with common methods in 
educational research. 

Common Qualitative Methods 

One of the most common types of 
educational inquiry is field research 
(literally, “research in the field” just 
like a journalist would be “in the 
field”). In other words, the researcher 
goes to a school to find out what is 
happening. Field research makes up 
the bulk of qualitative analysis (re- 
search based on understanding the 
nuances and details of a situation) 
in educational research. Qualitative 
research is not focused on counting 
or classifying events, but on capturing 
descriptive accounts of them. 

Many folks will use the term eth- 
nography interchangeably with field 
research, but in fact, ethnography is 
merely one form of field research. It 
is based heavily in understanding cul- 
tures different from ones own (which 
is why it s a common method among 
anthropologists). So, unless you re 
planning to study educational tech- 
nology in a vastly different culture 
than your own, you wont see it much 
in Ed Tech research. 

Qualitative analysis (research 
based on understanding the nuances 
and details of a situation) in educa- 
tional research can be broken down 
basically into two categories: observa- 
tions and interviews. (Yes, there are 
many subtleties in types of qualitative 


methods, but let s keep this simple.) 
Many researchers will use the term 
participant observation to describe 
their fieldwork. However, participant 
observation is actually a continuum of 
fieldwork from the “complete observ- 
er” at one end to the “complete par- 
ticipant” at the other. As you might 
expect, most research falls somewhere 
between the two. 

In education, the observer typically 
monitors or studies classrooms and 
the school environment, looking for 
patterns in the ways students behave, 
in how teachers present material, and 
so on. Any time you’ve had an outside 
researcher come sit or stand at the 
back of your classroom simply watch- 
ing, maybe taking a few notes, that 
person was likely conducting partici- 
pant observation, but at the observer 
end of the scale. 

Now let’s focus more on a combi- 
nation of participation and observa- 
tion, where the researcher not only 
observes what is happening, but actu- 
ally participates in it, too. A school 
principal may decide that she’d like to 
lead a few classes’ work in the school’s 
computer lab, giving the students an 
assignment to use the Web to research 
a particular topic, and then observ- 
ing how well they accomplish their 
task, taking notes on their work, and 
perhaps providing some overall group 
instruction along the way. This is par- 
ticipant observation. 

Yes, there is concern that by par- 
ticipating, the researcher ultimately 
changes the situation he or she is 
researching. This is inevitable, really, 
but the difference between valuable 
and poor participant observation is 
how important those changes are. 

The quality researcher will attempt to 
minimize any changes in the setting 
that might directly affect what is be- 
ing studied. In the example above, the 
principal should not help particular 
students more than others in their 
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More on Research Terminology 

The Web contains a wealth of information to help you learn more 
about reading educational research. Visit these sites for information 
and further links. 

American Educational Research Association: http://www.aera.net 

American Evaluation Association: http://www.eval.org/ 
EvaluationLinks/Collections.htm 

Education Resources Information Center: http://www.eric.ed.gov/. 
Use this newly refurbished site to find educational research and 
other such publications. 

American Sociological Association— Available Data Resources: 
http://www.asanet.org/student/data.html. This page provides 
a list of data resources for social research, many of which apply 
to education. 

Education Directorate of the American Psychological Association: 
http://www.apa.org/ed/ 


Web searches. That would be inter- 
fering in the outcome of the study. 
The fact that the principal is work- 
ing with (and observing) a group of 
students may also have an effect on 
their behavior that is not normally 
there — we hope they shape up a bit. 
But it probably wont interfere much 
with how well students conduct Web 
searches. The bottom line for the 
participant researcher is: be aware of 
yourself Don’t let your own opinions, 
preferences, or habits interfere in the 
outcome of the study. 

Participant observation by itself 
is often limited in how much one 
can understand simply by watch- 
ing. Other times, the researcher has 
to talk to the teacher or the students 
(what a concept!). In this case, he or 
she would be conducting interviews, 
where the most common types are 
IDIs (in-depth interviews) and focus 
groups. IDIs are used to ask questions 
of the subject (the person being inter- 
viewed) and to probe (ask follow-up 
questions based on his or her previous 
answers) for more information. Focus 


groups allow for the same kind of 
question-and-probe method, but they 
also allow the participants to react to 
each other’s statements. Focus groups 
can be quite valuable in that the par- 
ticipants themselves become, in es- 
sence, extra researchers, making com- 
ments, asking questions, and building 
on what others have said to create a 
complete picture of the topic at hand. 
One note of warning, though, is 
that focus groups are not a good idea 
when the topic is of a sensitive nature, 
any time when participants might not 
be comfortable sharing some of their 
experiences with a group. In such a 
case, IDIs are preferable; they should 
be kept confidential and treated 
anonymously when the researcher 
presents a report. It is common to use 
these two methods in combination. 

Common Quantitative Methods 

Although participant observation and 
interviews comprise major qualitative 
analyses in education, it is often nec- 
essary to provide numerical evidence 
in one’s research. Thus we come to 


quantitative analysis (research based 
on counting the number of times 
something happens.) Most of you are 
probably all too familiar with survey 
research, for it can sometimes feel like 
the educational community is being 
surveyed to death. Surveys can be 
broken down into four basic types: 

Mail surveys are those paper sur- 
veys you receive in the mail, fill out, 
and send back. They’re valuable in 
that you can reach just about any per- 
son, anywhere, as long as they have 
a mailbox. But they can also be ex- 
pensive, especially when your sample 
(the group of people being studied) is 
large: not only do you have to pay for 
printing, stuffing envelopes, and post- 
age, but you have to provide return, 
postage-paid envelopes if you expect 
to get any surveys back. Some kind 
of incentive, such as coupons or a 
prize drawing, can help increase your 
response rate (the percent of all the 
surveys you send out that you actually 
get back). 

Intercept surveys are also pencil- 
and-paper surveys, but are usually 
conducted by someone else on the 
respondent (the person answering 
the questions). That someone else 
holds a clipboard, asks questions of 
the respondent, and makes notes of 
his or her responses. If you’ve ever 
visited a mall during peak shopping 
periods, you’ve probably been asked 
at least once, by someone holding a 
clipboard, if they could have a mo- 
ment of your time. This person was 
likely conducting an intercept survey. 
Intercept surveys can be helpful if the 
topic is difficult to explain, because 
the person with the clipboard usually 
has a little bit of leeway in rephrasing 
questions that a respondent doesn’t 
understand. (They will likely have a 
set list of phrases or words that can be 
used to explain the question, so the 
surveys can be replicated.) 

Telephone and electronic or 
Web-based surveys leave the clip- 
board behind in favor of computers. 
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Both share the idea that answers to 
questions can be put directly into a 
computer for analysis, which can save 
a lot of time. Mail and intercept sur- 
veys must be keyed into a computer 
later, although this is even changing 
with intercept surveys as handhelds 
replace the clipboard. Unfortunately, 
telephone surveys are losing their 
effectiveness in the general public be- 
cause telephone solicitors and “push 
pollsters” have ruined the survey- 
takers reputation by using some un- 
ethical methods. Add in the fact that 
the public is generally inundated 
with unwanted phone calls of any 
type, and you can see why telephone 
surveys are losing their effectiveness. 

Web surveys may seem like a 
good alternative, because the invita- 
tion to take a survey can be e-mailed 
or placed on a Web site, allowing 
potential respondents to decide on 
their own time whether to partici- 
pate. Unfortunately, they, too, have a 
couple of shortcomings. First, e-mail 
invitations sent to people who aren’t 
expecting them could be considered 
spam. Second, not everyone has ac- 
cess to a computer with an Internet 
connection, or at least not necessarily 
at a time when it’s convenient to take 
a survey. So for now, Web surveys best 
serve the researcher when he or she 
has a predefined sample and when it 
is known that each person in the sam- 
ple has free and open access to a com- 
puter with an Internet connection. 

Researchers also use unobtrusive 
research (data collection that doesn’t 
require the researcher to bother any- 
one) to gather information. In educa- 
tion, the two most common types of 
unobtrusive research are existing data 
sources and content analysis. A typi- 
cal analysis of an existing data source 
would be pulling students’ test scores 
from across a school district and com- 
paring average scores across schools. 
Only one or two people have to be 
bothered with this, and they usually 
work in the district’s assessment or 


information technology departments. 
Content analysis is like existing data 
sources research in that the researcher 
is looking at something that already 
exists, such as textbooks, student 
artwork, or teaching portfolios. The 
researcher then analyzes the content 
of these artifacts (the things being 
studied in content analysis), looking 
for common patterns of language or 
expression, striking dissimilarities 
among artifacts, etc. 

Quasi-Experimental and Experi- 
mental Research 

Educational researchers have heard an 
awful lot about these last two types 
of research recently. In true experi- 
mental research, the researcher takes 
the subjects (those being studied, for 
example, students), randomly assigns 
them into two (or more) groups, 
puts them in similar surroundings 
with similarly trained teachers, and 
asks them at some specified point in 
time to take a test — let’s say, a math 
test. The difference between the two 
groups is that before taking the math 
test, the treatment group would be 
given the thing the researcher wanted 
to know about — new computers or 
new math software, for example — 
while the control group would learn 
math from the textbook or whatever 
preexisting instructional method was 
in use at the school. The new software 
or new computers are known as the 
treatment or the intervening vari- 
able. (Sometimes differing surround- 
ings or differently trained teachers are 
the intervening variables.) The goal, 
of course, is to see if the new software 
produced higher scores on the math 
test than simply learning from the 
textbook. 

But educational settings are rarely 
this neatly defined, and there is an 
ethical dilemma in providing one 
type of instruction or resource to 
one group but not to another. So 
we usually see quasi-experimental 
research in education. In such a case. 


two groups are still compared based 
on a treatment, but the students are 
not randomly selected and the treat- 
ment is often only available to one 
group. For example, if a school gets 
a grant for new computers, we might 
compare its average math test scores 
in each grade to the average scores in 
each grade from a similar school that 
did not get such a grant. If no other 
school is available for comparison, we 
might make comparisons over time 
using a pretest/posttest design. Sup- 
posing that the school’s fourth graders 
all learned math on new computers 
this year, we might compare their 
math test scores from the fall and 
spring when they were in third grade 
with their scores in the fall and spring 
this year. If their scores increased by 
5% in third grade, but by 15% over 
the course of fourth grade, we can 
conclude that the computers may 
have had an effect. Note that it is may 
have had not had, because it’s difficult 
to control for every other factor that 
might affect changes in math scores. 
Still, it’s reasonable to conclude that 
the computers had some effect, and 
our quasi-experimental design has 
yielded some worthwhile conclusions. 

With these tools on hand, it’s my 
hope that you’ll return to the RW 
column in future issues of L&L, pre- 
pared to gain some insight into how 
educational research affects your own 
work. Or at the very least, your new- 
found knowledge will make you the 
life of the party. 

Robert Kadel is the founder 
and a general partner of 
Kadel Research Consulting, 
LLC, located in Columbia, 
Maryland. His firm focuses on 
the evaluation of educational 
programs in technology, school 
reform, and community involvement. Robert has 
been working with computers for 25 years, since 
learning BASIC on an IBM mainframe in the 
1970s. With his graduate degrees in sociology 
and his focus on educational research, the evalu- 
ation of technology-supported education became 
a natural fit. 
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